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Abstract 



The purpose of this study is to explore aspects and indicators most commonly used to assess the 
quality of education systems in different countries through the comparison of 12 national 
publications describing the state of the educational system. 

To compare indicators the CIPP model was chosen. This model is organized in four main parts: 
Context, Input, Process and Product. Each main part was then divided into categories based on the 
research literature (e.g.. Input is divided into human, financial and material resources). 

Based on the CIPP model an electronic database for quality indicators comparison has been 
designed and implemented. This system is dynamic and easy to update. 

Using the database it is possible to compare countries according to several criteria (e.g., main parts 
and categories of the CIPP model, ISCED level, level of data collection). 

The database is available to the public and is designed for researchers and scholastic decision 
makers. 

A first comparison shows that processes at the classroom level are rarely considered, whereas 
public financial resources as well as achievement results are always included. 

The study concludes with a discussion of the findings of the comparison, as well as suggestions for 
future research aimed at further defining relevant indicators for the assessment of education quality. 
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Statistics, Databases. 




Introduction 



In many countries both in Europe and elsewhere it is customary to publish a summary framework 
outlining the state of the education system for the purpose of bringing to light information on the 
most important aspects of this system - key figures - and making this information accessible to the 
public, assembling material from different sources and at the same time providing clear and specific 
reading pathways. An examination of publications from 12 European and non-European countries 
with a view of assessing the quality of school systems by means of indicators has made it possible 
to compare the choices made by the various countries, be they methodological and conceptual 
(theoretical frames of reference, indicators, etc.), or technical and practical (type and frequency of 
publication, issues discussed, etc.). The project shows how each country, starting from processed 
information already present in an institutional data-base, has moved on to the construction of a 
wider theoretical framework capable of organizing the data, whether already collected or gathered 
ad hoc, in such a way as to allow the identification of aspects bearing on the description and 
assessment of the quality of scholastic systems in their various organizational configurations 
(national, local, single school, individual student). 

The aspects / indicators identified by each country in its own context permit both spatial and 
temporal comparisons. Comparisons over time show how different education systems have 
developed and changed and reveal any effects of reforms or other administrative measures. 
Territorial comparisons expose differences and similarities among the education systems of 
different countries and at the same time facilitate an analysis of the ‘distribution’ of the educational 
possibilities within each country at all levels (national, local, single school, individual student). The 
goals of Eisbon regarding education are an example of a joint temporal and spatial analysis. In 2004 
the Ministers of Education of the countries of the EU adopted a set of common objectives for the 
improvement of their education and training systems and a work program for achieving them, 
known as Education and Training 2010 (Council of the European Union, 2004). A constantly 
updated series of indicators and benchmarks was developed for monitoring the progress of each 
country toward the common objectives established. 

In conclusion, assessing the quality of school systems using indicators addresses the goals of 
making summary information about the most important aspects of the education system transparent 
and accessible to the public, while at the same time providing policy makers with objective 
evidence for evaluating the health of their country’s system of education and training. 




1. Assessing the quality of education systems using ‘systems of indicators’ 



The utilization of systems of indicators is today internationally regarded as the main tool for 
collecting objective information for the evaluation of school systems. These tools allow 
comparisons in time and space, making it possible to monitor changes in single phenomena 
observed over time and in specific contexts (different education systems, different geographical 
areas, different scholastic institutions, etc.). 



1.1. What is an indicator? 

In the field of education research, an indicator may be considered as a means of supplying 
information on the state of an educational system, a device that indicates whether it is working 
correctly or not (analogous to what happens on the dashboard of a car, where the various 
instruments allow the driver to check that everything is working properly). The indicator does not in 
itself say anything about the cause of a particular problem or point to a solution; it simply serves as 
a symptom that draws attention to one or more aspects of the education system’s condition of health 
(Castoldi, 1996). The definitions of indicators^ focusing on the education system, must be 
integrated with the perspective of the individual school in a context of school-based management - 
that is, of decentralization of the educational decision-making process through the involvement in 
the schools of both the parents and the community (International Bank for Reconstruction and 
Development, 2008). The metaphor of the dashboard of a car has often been utilized - and often 
referred to - in scholastic language. From a strategic point of view the use of indicators reinforces 
the ‘logic of the dashboard’, creating conditions in which the school can keep the efficiency and 
effectiveness of the curriculum under control (Romei, 1999). 

The indicators used by different countries are reducible to three main typologies: 

• In some cases countries simply supply information that doesn’t form a basis for any particular 
judgment: number of scholastic units, number of students, etc. Information like this is used to 
design a service, not to evaluate it. 

• In other cases, indicators are tapped to study certain aspects more thoroughly (such as 
incoming or ongoing training of teachers), i.e., they have a descriptive value. 

• Other indicators are instead linked to variables which are assigned a value or quality criterion 
- positive if the variable seems to promote learning and negative if it seems to represent an 
impediment (for example teachers’ transfer requests). In some cases it is enough to know if 

* For a more technical definition of indicators in a scholastic setting, see Oakes (1986) and Shavelson and McDonnel 
(1987) 




such a criterion exists or not (whether a laboratory is used for educational purposes, for 
example), but in most cases this is not enough - it will be important to know the degree to 
which a criterion exists and, above all, to what extent this is considered acceptable. 

Indicators are closely connected to the concept of standards. It is not enough to know how high or 
low a value is for a certain indicator, but it is also important how close such a value is to the 
standard established as reasonable in the situation. 

Standards can be: 

• minimum requirements such as those set by the authorities in each country (e.g., the minimum 
number of students per class); 

• benchmark standards, whose actual level is set against what is considered to be their 
theoretical level (e.g., grouping together schools of the same type, or students in the same 
socio-economic range); 

• standards concerning objectives - levels of excellence that can be set as goals. 



1.2. What is a system of indicators? 

Considering the complexity of the situation under analysis - that is, the educational system - it is 
usually insufficient to rely on a single indicator; it becomes necessary to put together a series of 
indicators, each of which is capable of focusing on one part of the system. Indicators are not simply 
juxtaposed measurements; they make up a coherent structure, a ‘system’ of data that can present a 
valid picture of a system of education. This is the sense in which the expression ‘system of 
indicators’ is preferable to ‘single indicator’. 

The continuing evolution of systems of indicators reflects the need to arrive at a list that is as 
sensitive as possible to the assessment requirements of different education systems. This process 
often coincides with a gradual decrease in the number of indicators themselves . The experience of 
many European and non-European countries , along with that of the INES {International 
Educational Indicators) project dating back to 1987"^ shows that even after decades of 
experimentation, publications concerning the quality of the education system and the schools are 



^ An example of this situation is the map of indicators published on the website of the Spanish Institute de Evaluacion, 
at 

http://www.institutodeevaluacion.mec.es/sistema_estatal_de_indicadores_de_la_educacion/indice_de_indicadores/ 
[Access date: April 2010] 

^ For example, a report has been published in France since 1991 in which the education system is described using 
indicators; in Spain a similar report has appeared since 2000; in New Zealand since 2006, while in England the quality 
of the education system and the schools has been under examination for almost a century. 

The INES Project is the most important international initiative involving the construction of a set of indicators 
capable of comparing the development of different school systems and assessing their effectiveness and qualitative 
depth. The project was developed in the annual publication Education at a Glance. 




still reworked and updated, implying a serious commitment to the continuous revision of conceptual 
frameworks and related structures for gathering information. 

Over the years there has been increased attention to educational results and the processes believed 
to be linked to them, and this has therefore been extended to the indicators required to measure 
them. The need for a comparison of various countries’ approaches to the observation of their own 
school systems has grown as well. 



1.3. Why a system of indicators? 

The use of a theoretical frame of reference or framework underpins the structure and selection of 
the ‘system of indicators’ inasmuch as it offers a justification of the choices made - an explanation, 
that is, of the connections between the features described by the indicators and the procedures and 
techniques of data collection. 

A system of indicators is not, or not completely, able to give definite answers regarding the 
direction and strength of the relationships between different aspects, but it offers the opportunity to 
explore these relationships and work out a complete picture of the effects and possible causes. As 
Fitz-Gibbon and Tymms (2002) put it, “An indicator system is only a step along the way to trying 
to understand what works, and how schooling can be improved. Consequently, some of our 
indicator systems include process variables such as descriptions of methods of teaching and 
learning [...]. Process indicators serve to generate hypotheses and most importantly, they stimulate 
discussion of teaching methods among staff in schools and as such are valuable. The important 
problems in trying to attribute cause and effect must, however, be continuously emphasised”. 
Amongst the variety of the indicators analyzed in the several countries considered, three main key 
points were focused. 

The first point is that in most of the more developed education systems the frameworks adopted are 
structured matching a system perspective with a school perspective, often integrating quantitative 
and qualitative techniques to collect data. 

The second point is related to the evaluation’s objectives, which are findable on a continuum having 
his poles in the two different concepts of ‘develop’ and ‘control’: the ‘develop’ approach focuses on 
the role that evaluation can play in changing the education system, in a perspective of operators’ 
involvement and constituting a real organizational learning; while the ‘control’ approach refers to 
the bureaucratic controls, that is the conformity with procedures and laws. 

The third point deals with the definitions of the aspects investigated. While there’s a wide 
convergence about dimensions and areas of research, more difficulties are to be faced up regarding 
the definition of those elements (indicators, aspects, variables, etc.) considered able to describe 




those dimensions/areas. The most common strategy to avoid this obstacle is to publish the 
conceptual frameworks, in order to make clear aspects to be investigated and procedures chosen for 
measuring school characteristics. 



2. Reports on the education systems of some European and non-European countries 

The education systems of several European and non-European countries were studied along with 
their respective reports. The aim was to understand how information is presented, which framework 
is defined and used, and what aspects were chosen to describe, present or report to the community 
the state of the education system. 

In some cases the reports are lengthy (Germany’s, for example), while others present only brief 
summaries (as in the case of Einland). Many of the reports refer to a set of indicators, but only a few 
of these are organized on the basis of a clearly illustrated framework (for example the reports of 
New Zealand and Spain). 

The data are almost always accompanied by graphs that are easy to read and understand, and are 
often supplemented with commentaries, international comparisons and information regarding more 
than just the education system. 

The frequency of publication varies among the different countries, e.g., Erance publishes more 
annual reports, while in Spain they are biannual. Some countries have been systematically 
producing reports based on indicators for many years, for others this is a recent achievement, and 
still others (like Einland) have thus far produced only a single publication. 

The following are brief descriptions of the reports examined up to now. 

• In Australia, in 2000 the Ministerial Council on Education, Employment, Training and Youth 
Affairs (MCEETYA) agreed to the development of a set of national key performance 
measures to ensure that the key indicators of the outcomes of schooling in Australia would be 
publicly available. Starting from 2000 the National report on schooling in Australia has been 
available yearly on the website http://www.mceecdya.edu.au/mceecdya/anr/. In 2008 the 
report provides informations about schools and student, teachers and teaching, resourcing, 
equity, students attendance, students national and international assessments. 

• In Denmark, an annual publication presents a series of key figures in tables and graphs 
showing development trends, and illustrates these using Danish figures complemented by 
equivalent figures from selected countries. Key figures for education in Denmark are updated 
continuously, and in some cases previously published data is adjusted. Facts and Figures 
2007 gives an overview of the Danish education system and a quantitative description of 




developments in most of the various fields of education. Most of the figures presented in the 
publication are also shown on the web site of the Danish Ministry of Education 
(www.uvm.dk). 

In England, an annual report presents evidence from inspections and regulatory visits 
undertaken by the Office for Standards in Education, Children’s Services and Skills (Ofsted). 
Evidence is taken from inspection activity across the full range of Ofsted’ s remit, including 
childcare, children’s social care, local authority services for children and provision for 
education and skills in schools, colleges and adult learning. The report draws upon the 
findings both of routine inspection visits and of the focused survey inspections through which 
inspectors collect more detailed evidence about subjects and aspects of provision in social 
care, education and skills. 

In Einland, Statistics Einland, the Einnish public authority specifically established for 
statistics, produces the vast majority of Einnish official statistics, including the field of 
education. The publication Education in Finland 1999 provids basic information on the 
regular education system and other forms of Einnish education, such as enrolment, educational 
staff, educational expenditure, graduation, level of achievement, transition from school to 
work and social outcomes. 

In Erance, the Ministere de I'Education Nationale every year publishes L’etat de I’Ecole de la 
maternelle a I’enseignement superieur ( available also in English), a brief analysis of the main 
features of the Erench education system. This is a collection of 30 indicators concerning costs, 
activities and results of the school system, going from nursery school right up to university. In 
addition, the Indicateurs generaux. Aide au diagnostic, au pilotage des academies et a la 
contractualisation has been published since 2000, as a means of monitoring internal variations 
within the Erench school system. These indicators supply statistical information on the 
functions and services of the education system of each of the territorial areas that make up the 
Erench scholastic system (academie). Their use facilitates the analysis of situations in the 
academie and contributes to the development and monitoring of territorial action plans. 

In Germany, the report Education in Germany 2008 presents the second comprehensive 
empirical account of the German education system. The volume analyses all stages of 
education ranging from early childhood education and care through the general education 
system, vocational education and training and higher education right up to the continuing 
education of adults. The report is based on a project which was funded by the Standing 
Conference of the Ministers of Education and Cultural Affairs of the Eander in the Eederal 
Republic of Germany and the Eederal Ministry of Education and Research. The report, the 




In the Netherlands, the Dutch Ministry of Education, Culture and Science presents yearly the 
statistical publication Key figures regarding the results and status of its policy areas in 
education, culture and science in the last five years. The publication Key figures 2004-2008 
contains information for each educational sector regarding participants, institutions, staff, 
outcomes and expenditures, pre-school and early childhood education, early school-leaving. 
Lifelong Learning, the labour market for teaching staff and Dutch education in the 
international perspective. 

In New Zealand, the website Education Counts has been developed to simplify access to 
quantitative education information for Ministry of Education clients. Education Counts has 
published a set of education sector indicators (http://www.educationcounts.govt.nz/indicators). 
In order to be more than an ad hoc collection of statistics, an explicit framework for education 
indicators was built. There are six domains to the indicator framework: education and learning 
outcomes, effective teaching, student participation, family and community engagement in 
education, quality education providers, and resources. 

In Spain, the Institute de Evaluacion, directly subordinate to the Ministerio de Educacidn, has 
the task of putting together the “State system of education indicators”. Beginning in 2000, 
indicators have been updated every year. Since 2006, one group of indicators, called priority 
indicators, have been updated annually, while the rest are updated every two years. In the 
complete edition of 2006 and in the successive 2009 edition, 38 indicators were published, 15 
of which were priority indicators. The dimensions according to which the indicators are 
grouped are: context, resources, schooling, processes and results. The reports and the map of 
indicators are available on the site of the Institute de Evaluacion 
(http://www.institutodeevaluacion.mec.es/publicaciones/index.php). 

In Sweden, the publication Descriptive data on preschool activities, school-age childcare, 
schools and adult education in Sweden 2006 gives an up-to-date and comprehensive picture of 
how pre-school activities, school-age childcare, schools and adult education are organised, 
including for example the number of pupils and of staff in different forms of schooling. It also 
gives an account of the expenditures of the different forms of schooling and of the results that 
are being achieved. It is based on the statistics supplied to the national monitoring system for 
the education sector. 

In Switzerland, the Confederation and the Cantons have committed themselves to a long-term 
program of education monitoring. In 2006 a pilot version of the education report was 




published, followed in 2010 by the first official version. The Report on the Swiss Education 
System 2010 is a compilation of present knowledge regarding the Swiss system of education. 
It contains data from the research, statistical and administrative sectors covering the entire 
education system, from kindergarten to continuing education. 

• In the United States, Congress has mandated that the National Center for Education Statistics 
(NCES) produce an annual report. The Condition of Education. The publication The Condition 
of Education 2009 presents 46 indicators. These indicators focus on participation and 
persistence in education, student performance and other measures of achievement, the 
environment for learning, and resources for education. The indicators present the most recent 
available data. In many cases the data are from the last two or three years and in some cases 
from previous years (2004, 2005, 2006). The website presents an integrated collection of the 
indicators and analyses published in The Condition of Education 2000-2009 
(http://nces.ed.gov/programs/coe/index.asp). Some indicators may have been updated since 
they appeared in print. 



3. The model selected 

The model chosen for reviewing and comparing the reports from the different countries in question 
is the CIPP model. Context, Input, Process, Product, (Stufflebeam, 1971; Stufflebeam & 
Shinkfield, 2007). The idea behind the model is simple - for a proper assessment of results (of a 
system, a program or a project), they must be linked to a preliminary assessment of the inputs, 
resources and processes activated in a determined context. 

The CIPP is seen not so much as a model in which the results are linked by a deterministic 
relationship with the other variables, but as a schema or conceptual approach that permits, at least 
on a logical level, the presentation of a complete picture of effects and possible causes. The CIPP 
supplies information to the different theories that attempt to explain the complex of relations that 
exist among various phenomena in the field of education. The CIPP model allows a vast number of 
conceptions to be taken into account concerning school quality, from the traditional economic view 
based on the productivity of the service (primarily oriented toward outcomes and their social 
impact, given determined inputs and existing contextual conditions), to the idea of educational 
development, based mainly on the study of processes at the school and/or class level as a means of 
improving outputs. 

Another way of using the model is to look at each element on its own, determining whether each 
indicator presents itself in a form that is ‘acceptable’ (or is at an ‘acceptable’ level). The OECD 




publication Education at a Glance (Organisation for Economic Co-operation and Development, 
2009), represents the most authoritative example of this conceptual approach to the education 
system. 



3.1 The dimensions of the model 
The four dimensions of the CIPP model are: 

• the Context the school operates in; 

• the Inputs - that is, the resources the education system and the single school units have access 
to; 

• the Processes in place - in other words, the activities undertaken by the school; 

• the Results obtained - immediate, medium- and long-term. 

Knowing the Context dimension is extremely important. This is what allows the school to adapt to 
local conditions. In general, the context is a structural fact that cannot be modified by educational 
action, although some situations may be considered ‘malleable’ to the extent that it is possible to do 
something about them, while others are “given conditions” that are harder to change and represent 
constraints to be taken into consideration. 

The indicators in this dimension have been framed in five macro-areas: 

• demographic and economic aspects of the population (e.g.. Levels of employment and 
unemployment. Gross domestic product per capita)', 

• schooling (e.g.. Proportion of children in school for each level of education. Educational level 
of the Dutch population)', 

• size and reach of the school system (e.g.. Number of institutions. Schools and pupils by types 
of school)', 

• socio-economic and cultural background of families (e.g.. Level of education. Occupation of 
parents, Children living in low income households)', 

• community involvement in a school (e.g.. Involvement of parents. Contribution of state and 
local governments to the implementation of additional services such as a dining hall. 
Institutional grants per participant). 

The Input dimension concerns the resources at a school’s disposal. Such resources may consist of 
human capital (personnel and students), economic factors (financing and available funds) and 
material factors (available facilities and equipment). 

This dimension includes information regarding: 




• schools (e.g., Average size of educational establishments. Average number of students per 
class); 

• economic resources (e.g., Public spending for education. Spending per student); 

• material resources (e.g., Number of computers per student); 

• human resources (e.g.. Number of teachers. Age of teachers); 

• students (e.g.. Students arriving at school early or late. Foreign students). 

Research into school effectiveness has over the years contributed to the identification of the 
processes most closely associated with results and thus to the students’ level of learning and 
scholastic success. Process indicators normally fall into two large groups: processes at the school 
level and those at the class level. It was decided to recognize the autonomy of a third macro-area 
which includes processes involving cooperation between schools and local communities. 

The macro-areas presently identified are: 

• processes at the level of school and territory (e.g.. Participation and involvement of the local 
community and families in the activities of the school); 

• processes at the school level (e.g.. Flours of teaching offered. Time spent on various tasks by 
the head of the institute. Presence of teacher work groups); 

• processes at the class level (e.g.. Small-group projects. Support for students with special 
needs). 

The results produced by education systems take on an importance both in themselves and in relation 
to the processes put in practice to bring them about, to the resources invested, and to the context, 
which either does or does not promote scholastic success. 

The Results dimension is divided into four macro- areas: 

• knowledge and abilities acquired by the students (e.g.. Results emerging from standardized 
tests); 

• level of education and scholastic achievement reached (e.g.. Proportion of students who 
graduate. Rate of school-leaving); 

• social results (e.g.. Reduction in the unemployment rate among young people. Equal 
opportunity of men and women in access to higher education, Education and labour market 
status of age bracket 20-24 as a percentage of total,); 

• perceived quality of service (e.g.. User satisfaction). 




4. The structure used to categorize the indicators 

As shown above, the framework adopted for the classification of the indicators developed by the 
different countries under study is the CIPP model, on the basis of which each indicator may be into 
a tree structure. Beginning with the ‘dimensions’ (Context, Input, Processes, Results) - within 
which ‘macro-areas’ were identified, in their turn containing ‘areas’ - we arrive at a detailed 
description of each indicator. 

For each indicator, several distinguishing features have been highlighted so as to show how the 
indicator was constructed, what it is meant to illustrate, and the interpretation of it adopted by the 
country that uses it. In particular, we have chosen to call attention to features such as: 

• the position of the indicator in the framework (the CIPP model); 

• a description of the indicator; 

• an explanation of its utility, its interpretation and the reason it is considered important by each 
country; 

• technical information that gives details for example about how it was calculated; 

• the ISCED ^ level of the indicator; 

• the level of presentation of the data - that is, the scale according to which the data are 
processed / presented (by geographical / institutional area, or at a national level for 
comparison of different years or for international comparisons); 

• the unit of data collection - that is, whether the data were gathered at the level of the single 
individual (student, parent, teacher), or at the level of the class or the school. 

In addition to these, some functional indications were included for each indicator to help trace its 
origin, such as: 

• the country that defined it; 

• its original name; 

• its original code (if available); 

• the macro-area and the area in the original framework (if identified by the country of origin); 

• the source, referring to the original document where it was found. 



^ ISCED (International Standard Classification of Education) is a standard created by UNESCO as an international 
system of classification for education. 




5. A dynamic database for quality indicators comparison in education 



The indicators studied have been organized in a dynamic public on-line archive (Dynamic 
Database) that can be accessed and explored and is structured following the CIPP model. 

A functioning database was designed in such a way as to promote the widest possible utilization of 
its contents^. The dynamic structure of the database allows constant updating, it can be expanded 
and upgraded in keeping with new or more recent publications. 

The dynamic database serves as a tool for comparing systems of indicators according to several 
criteria (e.g., main parts and categories of the CIPP model, ISCED level, level of data collection) 
and identifying the indicators most used by different countries to portray their own education 
systems. The database is available to the public and is designed for researchers and scholastic 
decision makers. 

In appendix are shown the on-line archive’s structure {main display, custom view, specific indicator 
file) and its functions (filtering and/or ordering the indicators). 



6. Comparing indicators from different countries - some results 

An initial overall reading of the main display of the on-line Database reveals the priorities used by 
countries such as Spain, the Netherlands, France and New Zealand in selecting aspects to include in 
official publications concerning the assessment of school system quality by means of indicators. 

For example, a look at the Context dimension for different countries shows that the macro-area 
Schooling (and the two sub-areas Schooling at each level of education and Education level in the 
adult population) is a high-priority aspect for all countries. By contrast, the macro-area Community 
involvement in schools is described using different areas by different countries - Spain uses 
indicators concerning Involvement, France those regarding Additional services, while for the 
Netherlands the area explored is Study grants. The indicators for Socio-economic and cultural 
background of families of students are widely used - they are in fact frequently correlated with 
student achievement. 

The macro-area Economic / financial resources is the one that mainly represents the Input 
dimension - in particular, the area Public spending for education is covered thoroughly by all the 
countries. Indeed, the indicator Total education spending relative to GDP is regularly used in 
international comparisons. The area Characteristics of schools is normally presented in publications 



® The home page of the archive of international indicators is http://valsisindpub.invalsi.it/ 




on education systems, with particular attention given to indicators concerning the number of 
students and teachers, data commonly used to produce ratios and relative measurements (e.g.. 
Student-teacher ratio). The Characteristics of students are described using numerous indicators, the 
most common being those linked to situations requiring specific attention, such as the number of 
foreign students or repeating students. 

Contrary to what might be expected, only Spain publishes data on computer equipment available to 
students - in general the different countries studied seem not to give much attention to Resource 
materials. 

The Process dimension is generally speaking the one least commonly investigated through the use 
of indicators; it is often the case that countries that identify aspects to explore in this dimension are 
those with assessment systems that are either external to the school or internal but structured. 
Spain, for example, develops indicators for the exploration of Teaching strategies, the School 
climate. Cooperation among teachers and Leadership/coordination styles. Some aspects of process 
are considered by the Netherlands with regard to Curriculum and didactic activity design (for 
example with the indicator Schools' autonomy in determining the content of their curriculum). 
Information in the Results dimension concerning the Level of education reached and Scholastic 
achievement, and concerning Direct results of education is found in the publications of all the 
countries considered; particularly Rates of school-leaving and Student levels of knowledge and 
ability explored on the basis of the results of standardized tests are always included both in 
territorial comparisons (international, national, local and by single school unit) and comparisons 
over time. Only the Netherlands also includes aspects regarding Results of equal opportunity policy. 
Some countries choose to develop aspects specifically linked to their own social context; New 
Zealand for example highlights the difficulties of its young people by exploring themes such as 
Youth suicide, maintaining that themes such as these have implications for both families and 
schools. 

One interpretation of the French indicators regarding ISCED levels highlights the concentration of 
attention on lower and upper secondary schools (ISCED levels 2 and 3) compared with primary 
schools (ISCED level 1). Eor secondary education, in fact, numerous indicators are included both 
for Input (especially under Characteristics of students and schools) and for Results. One of the 
most detailed studies in this dimension shows how the Erench system concerns itself much more 
with indicators of Scholastic achievement (23 indicators) than with those concerning Student levels 
of knowledge and ability (4 indicators), by contrast with other countries. New Zealand, Spain and 
the Netherlands, for example, have more or less the same the number of indicators of Scholastic 




achievement as they have for Student levels of knowledge and ability, the latter including results 
from the international tests PISA, TIMSS and PIRLS. 

Another key to interpretation concerns data collection units; it is notable that in all the countries 
studied most of the indicators are developed from databases already established by various 
ministries of education or institutes for evaluation (data collected at a centralized level). Indicators 
collected at the class level are much less frequent. 

In the case of data collected at the level of the individual, a comparison of various countries reveals 
a difference between French and Dutch publications on one hand, and those from Spain on the 
other. In the first two countries, information collected at the individual level almost always regards 
the students’ results, while Spain integrates these with information on students’ aptitudes and 
behaviour. 

This first review of results shows the potential of the online database. The goal for the future is to 
explore, expand and define new keys to interpretation, providing insights to policy makers and 
educators and, more generally, to the public. 




Appendix - The on-line database: structure and functions 



The home page of the archive of international indicators is http://valsisindpub.invalsi.it . 

The database is available to the public in Italian language. The English version is in progress. 



Fig. 1. Main page 
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Archivio Indicator! internazionali 

Sulla base del Iavoro dl raccolta e studio degll Indicatori utilizzati pervalutare i sistemi dl istruzlone nei divers! paesi analizzati, si sta 
strutturando un archivio Interattivo perpresentare gli indicatori In modo coerente con il quadro dl riferimento unitarlo adottato dal gruppo di 
ricerca. 

Gli indicatori sono stall ordinati secondo un'organizzazione gerarchica in 4 livelli: 

■ Dimensione 

■ Macro-area 

■ Area 

■ Indicatore (per esempio: Input. Student!, Caratteristiche degii student!, Percentuale di student! con cittadinanza non italiana) 

Da questa organizzazlone sara possibile passare a visualizzazloni 'personalizzate', per esempio ordinando/filtrando per nazione e/o per una 
specifica dimensione (ad esempio ’Input'). 

Nel complesso I'utente avra la possibilita di; 

■ visualizzare gli indicatori organizzati in apposite schede descrittive 

■ ordinare e filtrare in fase di visualizzazione gli indicatori 

■ ricercare in base al testo 

Saranno inoltre disponibili alcuni percorsi dl lettura predefiniti: ad esempio sara possibile visualizzare I'insieme degll Indicatori relativi agli 
obiettivi di Lisbona, oppure un altro insieme che comprende indicatori utili per I'elaborazione di un bilancio sociale, ecc. 
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As shown below the on-line archive can be browse in three ways: 



a. Main display 

In the main display a navigable webpage presents the four main dimensions of the CIPP model. For 
each dimension it is possible to expand the view in order to explore the complete contents, 
including all the macro-areas, areas and indicators it contains, following the tree structure defined 
by the model. 

It is possible to view one or more dimensions on the same page. 





Fig. 2. Main display 
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[ Pagina scaricata in 0.113 s«condi. ] 



b. Custom view 

In the section of the site labelled ‘Ricerca’’ {Research) the user can move from the default main 
display to custom views by filtering and/or ordering the indicators in the database according to a 
series of criteria: 

• the identification code for each indicator, inserted in the database through an open field - 
knowing the code of an indicator allows the user to identify it directly; 

• the dimension - choosing one of the pre-defined dimensions allows the display of all the 
indicators it contains; 

• the macro-area - the user may select one of the macro-areas already specified or perform a 
search based on a key word; 

• the area - it is possible to enter a specific word and then filter the indicators only for those 
areas that contain it (e.g., if the word “resources” is entered, indicators will be displayed from 
the areas Resources of schools. Economic / financial resources. Resource materials. Human 
resources)', 

• the name of an indicator - it is possible to enter a key word and view only the indicators that 
contain it (e.g., entering the word “spending” will show indicators such as Total spending for 



’ http://valsisindpub.invalsi.it/search.php 









education as a percentage of GDP, Total public spending for education. Education spending 
per student, etc.); 

• the ISCED level - indicators may be selected on the basis of one or more of the levels of 
education that the user wants to look at (e.g., level 1 - primary school, level 1 and 2 - primary 
and lower secondary school); 

• the data presentation level - the user can select indicators on the basis of the territorial level at 
which they were presented by the different countries (e.g., only indicators compared at the 
international level, or only indicators compared at the regional level); 

• the unit of collection - indicators may be selected on the basis of the unit for which the 
information was collected (e.g., for the individual or for the school). 



Fig. 3. Custom view 
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Per fare una ricerca puoi: 



1. selezionare una sola caratteristica (es, Nazione: Francia) 

2. selezionare piu caratteristiche insieme (es. Dimensione: Input; Macroarea: Risorse materiali; Livello ISCED: 2). Con 
questo tipo di ricerca e anche possibile scegliere in base a quale caratteristica ordinare le informazioni (pulsante 
Ordina). 

3. fare una ricerca con testo libero (es. inserendo il testo matematic nel campo Nome indicatore, vengono visualizzati 
gli indicatori Risultati in matematica nella scuola obbligatoria, Andamento del punteggi medi nelle competenze 
matematiche a 10 anni, ecc.) 



Ricerca 



Pulisci 



In addition, several pre-defined reading pathways will be put in place, for example, it will be 
possible to view the set of indicators related to the Lisbon objectives, or a set that includes 
indicators helpful in the development of a school progress report. 







c. The specific indicator file 



However the contents of the database are displayed, whether it be the main or the personalized 
display, it is possible to select an individual indicator and open a file containing the following 
details: 

• Id - a unique identity code for each indicator. 

• Country - the country that defined it. 

• Dimension - the dimension under which each indicator is classified by the CIPP model. 

• Macro-area - the macro-area in which each indicator is entered according to the reference 
framework. 

• Area - the area in which the indicator is entered according to the CIPP model. 

• Indicator name - the name of the indicator translated into Italian. 

• Original name - the name of the indicator in the original language. 

• Description - a brief description of the indicator (e.g., percentage of the adult population 
between 25 and 64 years of age who have completed a certain level of education). 

• Explanation - introductory or explanatory comments on the use and interpretation of the 
indicator (e.g., this indicator presents data on young people with a school education between 0 
and 29 [...]). 

• Technical information - any available technical information on the construction of the 
indicator, whether from a document or the original website (e.g., the definition of levels of 
education lower than primary school refers both to citizens who are illiterate and to those who 
know how to read and write but have not had at least 5 years of schooling [...]). 

• ISCED level - the international codes that indicate the level of education that the indicator 

refers to. The ISCED levels are: 0 - pre-primary school, 1 - primary school, 2 - lower 
secondary school, 3 - upper secondary school, 4 - post-secondary training (non- 

university), 5 - tertiary education (university degree) 6 - advanced tertiary education 
(doctorate). 

• Data presentation - the level at which the data are processed / presented. This can be 
international if the data are used for comparison with similar data from other countries (e.g.. 
Public spending for education), national if the data are presented in aggregate for the whole 
country (e.g.. Number of institutions), by territorial / institutional divisions (e.g.. Number of 
students in each region) if the data are presented and compared following the organization of 
a country’s internal territorial or administrative units. An indicator can be presented in several 
ways at the same time. 




• Units of collection - the physical unit used in data collection. Data present in existing 
databases can be collected at a centralized level (e.g., Gross domestic product per capita), at 
the level of the school (e.g., Profile of the school principal), at the class level (e.g., Students 
per class), and/or at the level of the single individual (e.g.. Mathematics results from a 
standardized learning test). 

The same file goes on to present data that allow the user to trace the indicator to its original 
document: 

• Document source - the source (document or website) where all the information in the file was 
found. 

• Data source - the official source of the data used in the creation of the indicator (if available). 

• Code - the original reference code used by the country (if available). 

• Macro-area - origin - the macro-area where the indicator is classified by the country that 
defined / used it (e.g.. New Zealand classifies the indicators in 6 macro-areas: Education and 
learning. Student participation. Family and community, [...]). 

• Area - origin - the area where the indicator is classified by the country that defined / used it 
(e.g., Spain puts the indicators for schooling in different areas, such as Schooling at each 
educational level. Schooling and population. Evolution of school attendance rates [...], 
Access to higher education. Foreign students. Recognition of diversity [...], Participation in 
lifelong learning). 




Fig. 4. The specific indicator file 
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Nome Indicatore 


Spesa annuale per studente 




Nome originale 


Annual expenditure per student 




Descrizione 


Spesa annuale per studente nella scuola primaria e secondaria (marzo 2010). 




Spiegazione 


Un’analisi quantitativa effettuata utilizzando dati provenienti da scuole di lingua inglese 
(DFES, 2001)j ha fornito prove di una positiva e statisticamente significativa relazione tra 
gli investimenti di capitale e i risultati degli alunni. 

La spesa pubblica per studente e di particolare importanza quando la situazione socio- 
economica della famiglia dello studente o della comunita scolastica e bassa. Maggiore 
risorse per studenti sono necessarie per superare le barriere di apprendimento connesse 
con I'accesso alle risorse ed alle tecnologie dell'informazione e della comunicazione (TIC), 
e fornire opportunita legate al capitale culturale e scolastico. 

Una scuola efficace richiede la giusta combinazione di formazione e di talento personale, 
attrezzature adequate e moderne e studenti motivati pronti ad imparare. La domanda di 
un insegnamento di alta qualita, che puo tradursi in un aumento dei costi per studente, 
deve essere bilanciato rispetto alia richiesta di oneri eccessivi per il contribuente. 

Anche se e difficile valutare il volume ottimale delle risorse necessarie per preparare ogni 
studente pertutta la vita e per il lavoro nella societa moderna, i raffronti internazionali di 
spesa per I'istruzione per studente sono in grado di fornire un punto di partenza per 
valutare I'efficacia di diversi modelli di insegnamento. 




Note tecniche 


Numeratore: spesa annuale totale per la scuola primaria e secondaria usando la parita 
del potere d'acquisto per il PIL. 

Denominatore: numero totale di studenti della scuola primaria e secondaria full-time. 
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Fig. 5. References to the original document 
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